1. Field of the Invention
The present invention relates, generally, to device access fairness, and in particular embodiments, to control of device access fairness in a System Packet Interface (SPI) attached switch enclosure.
2. Description of Related Art
As illustrated in the exemplary interconnection diagram of a storage system shown in FIG. 1, non-blocking frame-based crossbar switch enclosures (e.g. frame-based switch enclosures 100 and 102) enable a “fabric” interconnection of a large number of devices such as Host Bus Adapters (HBAs) 104 and 106, and groups of disk drives referred to as Just a Bunch Of Disks (JBODs) 108, 110, 112 and 114. The devices are connected to ports in the frame-based switch enclosures 100 and 102. In FIG. 1, HBA 104 is connected to Port 1 on frame-based switch enclosure 100, JBOD 108 is connected to Port 2, JBOD 110 is connected to Port 3, and frame-based switch enclosure 102 is connected to Port 4.
Note that unlike Fibre Channel (FC) arbitrated loop (AL) storage switch enclosures, which utilize an 8-bit Arbitrated Loop Protocol Address (ALPA), have a 126 device limit, connect and switch loop devices that must share the bandwidth, and cannot be connected to any other storage switch, frame-based switch enclosures utilize a 24-bit address (which includes 8-bit domain and area fields in addition to an 8-bit ALPA), have a much higher device limit, and connect and switch devices that do not have to share the bandwidth. Frame-based switch enclosures also support loop devices, such as disk drives in a JBOD connected via a port, and can also be connected to other frame-based switch enclosures via an inter-switch link.
Although the exemplary frame-based switch enclosures 100 and 102 in FIG. 1 have four ports each for purposes of illustration only, it is often desirable to utilize frame-based switch enclosures with larger numbers of ports. To facilitate frame-based switch enclosures with larger numbers of ports, a number of Application Specific Integrated Circuits (ASICs), each ASIC containing a frame-based switch and referred to as a Switch On a Chip (SOC), may be cascaded to provide a greater number of ports within a frame-based switch enclosure.
As illustrated in the example of FIG. 2, SOCs 200 identified as SOC0, SOC1 and SOC2 are connected within a frame-based switch enclosure 202 using a System Packet Interface (SPI) bus daisy chained to SPI input and output ports on each SOC 200. It should be understood, however, that any number of SOCs could be employed within the frame-based switch enclosure 202, subject to space considerations within the enclosure. SPI is a multi-channel protocol time-division multiplexed (TDMed) over a single bus 204. The SPI bus 204 includes a control line, 16 data lines, and a clock. Conventionally, one virtual channel is assigned for every port in the frame-based switch enclosure 202, although other virtual channel assignment schemes may be employed. In the example of FIG. 2, each SOC 200 also contains four local ports identified as P0, P1, P2 and P3. It should be understood, however, that any number of local ports could be employed within the SOC 200, subject to space considerations within the SOC 200. Connected to the local ports of each SOC 200 can be one or more “don't care” Bunch Of Disks (xBODs) 206, which may be JBODs or Switched Bunch of Disks (SBODs), and one or more HBAs 208 or other devices.
Frames of data may be transferred over the SPI bus in bursts 300 (e.g. 64 bytes), illustrated symbolically in FIG. 3. At the beginning of each burst 300 is a control word 302 which contains the channel IDentifier (ID) 304 and may also contain a Start Of Frame (SOF) indicator 306, an End Of Frame (EOF) indicator 308, and parity information 310. Note that each burst 300 is but a portion of a larger frame 312 (e.g. 2048 bytes).
FIG. 4 illustrates an exemplary four port SOC 400 corresponding to the SOCs illustrated in FIG. 2. In the example of FIG. 4, SOC 400 contains four local ports 402, identified as Port0, Port1, Port2 and Port3. In addition, SOC 400 contains five physical ports 404 within SPI logic 412. Each physical port 404 may be programmably mapped to a virtual channel. A non-blocking crossbar switch core 406 is capable of being configured to make connections between the four local ports 402 and the five physical ports 404. SOC 400 also contains a processor 408 and a router 410 for configuring the switch core 406. Within SPI logic 412 is a transmit buffer bank 414 connected to an SPI output port 416, a receive buffer bank 418 connected to an SPI input port 420, and a pass-through First-In-First-Out buffer (FIFO) 422 connected between the transmit buffer bank 414 and the receive buffer bank 418. Note that in the example of FIG. 4, both the transmit buffer bank 414 and the receive buffer bank 418 contain one buffer for each virtual channel in the frame-based switch enclosure within which the SOC 400 resides. However, the buffer banks may be programmable to handle a different number of virtual channels. Each buffer may be able to store one or more bursts. The receive buffer bank 418 is connected to the five physical ports 404 at 426, which are connected to the transmit buffer bank 414 at 424.
In the case where a burst of data is to be sent from Port0 to Port3, for example, a request to send the data is first received at Port0. When sending data between local ports the D_ID (destination identifier) of a FC frame is used to “route” the frames between ports. If the request is granted (i.e. the ports needed to make the connection are available), the burst of data is routed through switch core 406 by the router 410, and transmitted at Port3 without involvement of the SPI logic 412.
In the case where a burst of data is to be sent from Port1 to SPI output port 416 for transmission to another SOC, for example, a request to send the data is first received at Port1. If the request is granted (i.e. the ports needed to make the connection are available within the SOC 400), the data is received at Port1, and the router 410 configures the switch core 406 such that the data is routed through a physical port 404 to transmit buffer bank 414 where it is stored in the appropriate virtual channel buffer. The stored data may then be transmitted at SPI output port 416.
In the case where a burst of data is to be sent from SPI input port 420 to Port2, for example, a request to send the data is first received at the SPI input port 420. If the request is granted (i.e. the ports needed to make the connection are available and ownership of the virtual channel is available within the SOC 400), the data is first stored in the appropriate virtual channel buffer of the receive buffer bank 418. The router 410 then configures the switch core 406 such that the data is routed from the receive buffer bank 418 through the physical port 404 mapped to the particular virtual channel associated with Port2 and over to Port2, where it is output.
In the case where a burst of data is to be sent from SPI input port 420 to SPI output port 416, for example, a request to send the data is first received at the SPI input port 420. If the request is granted (i.e. the ports needed to make the connection are available and ownership of the channel is available within SOC 400), the data is first stored in the appropriate channel buffer of the receive buffer bank 418. The data is then sent to pass-through FIFO 422 and on to the appropriate virtual channel buffer of the transmit buffer bank 418, where it is output.
As the foregoing examples suggest, while data from a local port (i.e. Port0-Port3) is being sent to the SPI output port 416 (i.e. while a local port has ownership of the virtual channel), data from the SPI input port 420 on the same virtual channel and destined for the SPI output port 416 (i.e. data utilizing the pass-through FIFO 422) cannot be transmitted, because the same virtual channel buffer in transmit buffer bank 414 is needed for both paths. While data from a local port is being sent to the SPI output port 416, data from the SPI input port 420 on the same virtual channel fills up the appropriate channel buffer in the receive buffer bank 418. Once that channel buffer is full, a feedback signal is sent back to the other SOCs, instructing them to temporarily suspend the transmission of data on that channel.
Because it is desirable to send an entire of frame of data through a local port before giving up ownership of the virtual channel, data in a channel buffer may have to wait a long time before it can be sent. Similarly, while data being received by the SPI input port 420 is being sent to the SPI output port 416 (i.e. while the pass-through FIFO 422 has ownership of the virtual channel), data from a local port on the same virtual channel and destined for the SPI output port 416 cannot be transmitted. Data may still be transmitted on other virtual channels. That is, if one channel is blocked and waiting for ownership, other channels may still transmit data to the SPI bus. To ensure that repeated data requests from a local port do not prevent data requests from the SPI input port 420 from being processed, or vice versa, a simple conventional fairness scheme alternates the processing of frames of data on the same virtual channel between a local port and the SPI input port 420.
However, even though frames of data on the same virtual channel from local ports and the SPI input port take turns being transmitted through SPI output port, a starvation problem can nevertheless occur when multiple SOCs are cascaded within a frame-based switch enclosure.
FIG. 5 illustrates an exemplary frame-based switch enclosure 500 with four SOCs 502, identified as SOC0, SOC 1, SOC2 and SOC3. For simplicity of explanation, each SOC in FIG. 5 has one local port identified as P0. As mentioned above, one virtual channel is assigned to each port within a frame-based switch enclosure. In the example of FIG. 5, P0 of SOC0 is connected to disk drive 0 (D0) and assigned virtual channel 1 (SOC1), Port0 of SOC1 is connected to an initiator 504 (e.g. an HBA) and assigned virtual channel 2 (SOC2), P0 of SOC2 is connected to disk drive 2 (D2) and assigned virtual channel 3 (SOC3), and P0 of SOC3 is connected to disk drive 3 (D3) and assigned virtual channel 4 (SOC4). In the example of FIG. 5, drives D0, D2 and D3 are all sending data back to initiator 504 using CH2.
In the example of FIG. 5, because the initiator 504 is not sending data to any other device, no data bursts appear on SPI output port 506 of SOC1. Because D0 is sending data to initiator 504, frames of data from D0 (see reference character 508) must be sent to SPI output port 510 of SOC0. In addition, because no data is arriving at SPI input port 512 of SOC0, no alternating of frames is necessary in SOC0, and SPI output port 510 will repeatedly output the frames of data from D0 (see reference character 514). In SOC3, data from D3 must be sent to initiator 504 through SPI output port 516, and also the data arriving at SPI input port 518 from SOC0 must be sent to initiator 504 through the same SPI output port 516. Using an alternating frame fairness scheme, SPI output port 516 will alternate outputting frames from D3 (see reference character 520) and frames from SOC0 (which are the frames from D0; see reference character 522). In SOC 2, data from D2 must be sent to initiator 504 through SPI output port 524, and also the data arriving at SPI input port 526 from SOC3 must be sent to initiator 504 through the same SPI output port 524. Using an alternating frame fairness scheme, SPI output port 524 will alternate outputting frames from D2 (see reference character 528) and frames from SOC3 (which are the frames from D3 and D0; see reference character 530). Because frames from SOC3 alternate between frames from D3 and D0, SPI output port 524 transmits frames from D3 and D0 only once every four frames, as compared to frames from D2 which are transmitted every other frame. Thus, devices that are “upstream” from the destination port are starved as compared to devices that are closer to the destination port.
Therefore, there is a need to improve the access fairness of upstream devices in SPI attached frame-based storage switch enclosures.